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Abstract 

Given an i.i.d sample (Yi,Zi), taking values in ~R d x M. d , we consider a collection 
Nadarya- Watson kernel estimators of the conditional expectations E(< c g (z), g(Y) > 
+d g (z) | Z — z), where z belongs to a compact set H C R d , g a Borel function on 
R d and c g (-),<i g (-) are continuous functions on R d . Given two bandwidth sequences 
h n < t)„ fulfilling mild conditions, we obtain an exact and explicit almost sure limit 
bounds for the deviations of these estimators around their expectations, uniformly in 
g € Q, z € H and h n < h < tj„ under mild conditions on the density fz, the class 
G, the kernel K and the functions c g (-),d g (-). We apply this result to prove that 
smoothed empirical likelihood can be used to build confidence intervals for conditional 
probabilities P(Y € C | Z = z), that hold uniformly in z € H , C € C, h G [h n , t) n }. 
Here C is a Vapnik-Chervonenkis class of sets. 
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1 Introduction and statement of the main results 



Consider an i.i.d sample (Yi, Zi)i=\ n taking values in M d 'xR d , with the same distribution 
as a vector (Y,Z), and write < •, • > for the usual inner product. In this paper, we 
investigate the limit behaviour of quantities of the following form (assuming that this 
expression is meaningful): 



W n (g,h,z) ^fziz)- 1 ' 2 ^ ( < Cg(z),g(Yl) > +d g (z))K^- 

i=l - 

-e(( < Cg(z),g(Yl) > +d g {z))K{^^\ 



(1) 



Here, K denotes a kernel, h > is a smoothing parameter, g is a Borel function from R^ 
to R fc and fz is (a version) of the density of Z. Given a class of functions Q satisfying some 
Vapnik-Chervonenkis type conditions (see conditions (HG1) below), and given a compact 
set H, Einmahl and Mason (2000) showed that somewhat recent tools in empirical processes 
theory could be used efficiently to provide exact rates of convergence of 

sup { | W n (g, h n , z) |, g £ Q, z £ H}, 

along a bandwidth sequence h n fulfilling some mild conditions (see condition (HV) in the 
sequel). The exact content of their result is written in Theorem Q] below. The contribution 
of the present paper is twofold. As a first contribution, we provide an extension of the 
result of Einmahl and Mason, by enriching Theorem [T] with a uniformity in the bandwidth 
h, when h is allowed to vary into an interval [h n , f) n ], with h n and fj n fulfilling conditions of 
Theorem [TJ This extension is stated in Section 11.21 (Theorem [2]) , and is proved in Section 
[3l As a second contribution (Theorem [3]) , we apply our Theorem [2] to establish confidence 
intervals for quantities of the form 

Y € G I Z = z), C EC, z £ H, 



by empirical likelihood techniques. Indeed, we prove that these confidence intervals can be 
built to hold uniformly in z 6 H, C £ C and h £ [h n , t) n ], under conditions that are very 
similar to those of Theorem [2j This result is stated in Section \1. 41 and is proved in Section 
® 

1.1 A result of Einmahl and Mason 

As our first result is an extension of Theorem 1 in Einmahl and Mason (2000) we have 
to first introduce the notations and assumptions they made in their article. Consider a 
compact set H C M. d with nonempty interior. We shall make the following assumption on 
the law of (Y,Z). 

(Hf) (Y, Z) has a density fy,z that is continuous in x on M. d X O', where O' C M. d 
is open and where H C O'. 

Moreover fz is continuous and bounded away from zero and infinity on O' . 
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From now on, O will denote an open set fulfilling H C O C O'. Now consider a class Q of 
functions from M d ' to M k . For Z = 1, . . . , k, write Gi := IT; (^?) , where IIj(a;i, . . . , xi, . . . , x^) '■= 
xi for (xx, . . . ,x k ) G R fc . 

(HG) Each class is a pointwise separable VC subgraph class and has a finite valued 
measurable envelope function Gi satisfying, for some p G (2, oo]: 
a := max l= i r .. ifc sup 2SO || Gi{-) \\c Y \ z=z , P < oo, 

where || Gi(-) \\c Y \z =1 .,p ^ s the L p -norm of Gi under the distribution of Y Z = z. For a 
definition of a pointwise separable VC subgraph class we refer to Van de Vaart and Wellner 
(1996, p. 110 and 141). Now, for any g G G, consider a pair of functions (c g (-),d g (-)), where 
c g maps W d to M. k and d g maps M. d to M, and assume that 

(HC) The classes of functions T>\ := {c g , g G G} and T>2 '■= {d g , g G G} are uniformly 
bounded and uniformly equicontinuous on O. 

We now formulate our assumptions on the Kernel K, with the following definition. 

K:={k(\--z), A>0, (2) 

(HK1) K has bounded variation and the class K, is VC subgraph. 
(HK2) K(s) = when s <£ [-1/2, l/2] d . 
(HK3) f K(s)ds = 1. 

R d 

Note that (HK1) is fulfilled for a quite large class of kernels (see, e.g., Mason (2004), 
Example F.l). In Einmahl and Mason (2000), the authors have studied the almost sure 
asymptotic behaviour of 

sup{ | W n (g,h n ,z) |, g G G, z G H) 

(recall ([I])), along a bandwidth sequence (/i n ) n >i that satisfies the following conditions 
(here we write log 2 n := log log(n V 3)) : 

(HV) h n | 0, nh n d too, log(l//i n )/log 2 n ^ oo, h d (n/ \og(l/h n )) 1 ~ 2/p oo, 

where p is as in condition (HG) . We also set 

A 2 (g,z) :=E(( < c g (z),g(Y) > +d g (z)) 2 \z = z G g G 0, (3) 

A 2 ( 5 ) :=supA 2 ( 5 ,z), 9 e5 (4) 

A 2 (g):=supA 2 ( 5 ). (5) 

see 

Given a measurable space (x,T), a measure Q an d a Borel function i/> : \ ^ ^> we write 

\\^\\q, p = / \V\dQ. (6) 

x 

Under the above mentioned assumptions, Einmahl and Mason have proved the following 
theorem, A denoting the Lebesgue measure. 
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Theorem 1 (Einmahl, Mason, 2000) Under assumptions (HG), {HC), {Hf), (HK1)- 
(HK3) and (HV), we have almost surely 

urn sup = A(g) || K || A; 2 ■ (7) 



2nh d n \og(K' 



We point out that ([7]) is slightly stronger than Theorem 1 of Einmahl and Mason (2000) , 
as fz{z)~ 1 / 2 appears in our definition of W n (g,h, z) which is not the case in their paper. 
However, ([7]) is a consequence of their Theorem 1, as f z is uniformly continuous on H, 

by (Hf). 

1.2 An extension of Theorem [1] 

Our first result states that Theorem Q] can be enriched by an additional uniformity in 
h n < h < \] n in the supremum appearing in ([7]), provided that (h n ) n >i and (f) n ) n >l do 
fulfill assumption (HV). We also refer to Einmahl and Mason (2005), where the authors 
provided some consistency results for kernel type function estimators that hold uniformly 
in the bandwidth (see also Varron (2008) for an improvement in the case of kernel density 
estimation). 

Theorem 2 Assume that (HG), (Hf), (HC) and (HK1) - (HK3) are satisfied. Let 
(h n )n>i an d (f)n)n>i be two sequences of constants fulfilling (HV) as well as h n = o(t) n ). 
Then we have almost surely 

v I W n (g,h,z) | 

hm sup —j^^^=^^^= = A(y) \\ K \ \\2 ■ (8) 

eH, g eg, h n <h<t) n V 2nh d log(/i^ d ) 



n— >oo 



The proof of Theorem [2] is provided in Section [3 

Remark 1 Einmahl and Mason (2005) have proved a result strong enough to derive that, 
under weaker conditions than those of Theorem^ we have almost surely 

v fz(z) 1 / 2 W n (g,h,z) 

limsup sup : < oo. (9) 

rwoo zeH,geg, y nh d log(l/h) + log log n 

fte[^p,i] 

However, the finite constant appearing on the right hand side of ([Pj) is not explicit in their 
result. The main contribution of Theorem is that the right hand side of (OJ) is explicit, 
by paying the price of making stronger assumptions. 

Remark 2 As Theorem [D is an extension of Theorem [I] of Einmahl and Mason, all the 
corollaries of Theorem[J\ (see Einmahl and Mason (2000)) can be enriched with a uniformity 
in the bandwidth. 
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1.3 Some applications of Theorem [2] to data-driven bandwidth selection 

The main statistical interest of Theorem [2] is that we can derive the limit behavior of 
kernel regression estimators with data-driven bandwidth. Let us consider such a random 
bandwidth h n (z) = h(z, Y\, . . . , Y n , Z±, . . . , Z n ) that depends on the sample as well as on 
the point z £ M. d . In the sequel, Id shall denote the identity function. Our next corollary 
gives the a.s. limit behavior of the Nadaraya- Watson estimator 



r n {z) = V V 7 ' r Yi 



of the regression function r(z) : = E(Y | Z = z), when h n (-) satisfies some mild conditions. 
Note that the asymptotics are given for r n (-) — r(h n ,-), with 

^ f / — 2J\ 

r(h n ,z):=h„ / V K \~y i — ) fr,z(y,u)dudy. 

R d xR fe 

The random differences r(h n , z) — r(z) can be controlled by analytic arguments as soon as 
the a.s. limit behavior is known. 

Corollary 1 Assume that h n {-) satisfies almost surely (resp. in probability) 

tog(lAn) lQg(lAn) ^ 
< hmmf < hmsup — < 1. 

rwoo logn n^oo log n 

Then, we have 



±V 'fz(z){r n (z) - r(h n , z))Jnh n (z) d 

limsup sup . =|| K \\\2i 

n— >oo zeH J2A(Id,z)log(h n (z)- d ) 



almost surely (resp. in probability). 

Proof: The proof involves continuity arguments for A(Id, •) and the fact that the numer- 
ator and denominator of r n (z) are specific forms of the general object W n appearing in 
Theorem [2J We also consider the countable collection of events 

|n" r <h n < n~ r ' for all large n j, r,r'eQn (0, 1). 

On each of these countable events, the sequence h n can be bounded from below and above 
by sequences h n and \) n fulfilling condition (HV). We omit technical details. □ 

Example 1: Tsybakov's plug-in selection rule: 

Tsybakov (1987) considered a plug-in bandwidth selection rule when d = k = 1. In that 
case, he suggested that, for a given point 26I, the bandwidth should be chosen of the 
form 

h n (z) := ^(z)n- 1 / 5 , 
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where (3 n (z) is a consistent estimate of the theoretical quantity /3(z) that minimizes the 
asymptotic square error of r n (z). Under the conditions stated in Tsybakov (1987), since 
most of them being consequences of the assumptions of Theorem El the plug-in bandwidth 
satisfies the assumptions of Corollary [TJ 

Example 2: cross validation: 

We again consider the case d = k = 1. An important example is the bandwidth h n that 
minimizes the sample-based quantity 

n 

CV(h) := -YFi-T^ZiffwiZi), h G [n- 1+ *,n- s ], 
i=i 

where w is a weight function on R and 5 > is a fixed (small) value. We refer to Clark 
(1975) and Priestley and Chao (1972) for more details on that technique. By construction 
the random sequence h n satisfies the assumptions of Corollary [TJ Moreover, it is shown in 
Hardle et al. (1988) that, under mild conditions, we have h n ~ Con -1 / 5 in probability, for 
a theoretical constant Cq. 

1.4 Asymptotic confidence bands by empirical likelihood 

Empirical likelihood methods in statistical inference have been introduced by Owen (2001). 
This nonparametric technique has suscitated much interest for several practical reasons, the 
most important one being that it directly provides confidence intervals without requiring 
further approximation methods, such as the estimation of dispersion parameters. Moreover, 
empirical likelihood is a very versatile tool which can be adapted in many different fields, 
for instance in estimation of densities or conditional expectations by kernel smoothing 
methods. The idea can be summarised as follows : consider an independent, identically 
distributed sample (Yl, Zi)\<i< n taking values in R d ' x R d . Given h > 0, z € H, a function 
g from M d ' to M k and a (kernel) real function K, define the following centring parameter, 
which plays the role of a deterministic approximation of W,(g(Y) \ Z = z) : 

eUy)k{^)) 

m(g,h,z):= V (10) 

^(^)) 

This quantity is the root of the following equation in 9: 

^ K [^){g{Y)-e)^ =0, (11) 

which naturally leads to the following formula for a confidence interval (around m(g, h, z)) 
by empirical likelihood methods (for more details see, e.g., Owen (2001), chapter 5) : 

I n (g,h,z,c) := {9 € R, K n (9,g,h,z) > c}, (12) 
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where c £ (0, 1) is a given critical value that has to be chosen in practice, and where 

i=l i=l i=l 



n n ^ n 

K n (9,g,h,z) : = max ^YlnPi,J2Pi K {- 1 Y^){9{Yi)-e) =0, Pi >0, = l}. 



(13) 

It is known (see, e.g., (2001), chapter 5) that, for fixed z £ M. d and fixed g, we can expect 

m(g, h, z) £ I n (g, h, z, c) (14) 

to hold with probability equal to ¥(x 2 < —2 log c), ultimately as n — > oo, h — > 0, nh d — > oo 
(see e.g., Owen, chapter 5). A natural arising question is: 



• Can we expect (|14p to hold uniformly in z, g and h? 

• In that case, how much uniformity can we get? 

Uniformity in g and z would allow to construct asymptotic confidence bands (instead of 
simple confidence intervals), while a uniformity in h would allow more flexibility in the 
practical choice of that smoothing parameter. Our Theorem 3 provides a tool strong 
enough to give some positive answers to these questions. We shall focus on the case where 
Q = {lei C £ C} for a class of sets C. We will also make an abuse of notation, by 
identifying C and Q, and hence, we shall write m(C, h, z) for m(lc, h, z) and so on. Write 
the conditional variance of lc(Y) given Z = z as follows : 

a 2 {C,z) :=F(Y eC\ Z = z) - F 2 (Y GC\ Z = z) , C G C, z e H. (15) 

The next theorem shows that we can construct, by empirical likelihood methods (recall 
(TT2Tl ) . confidence bands around the centring parameters m(C,h,z) with lengths tending 



to zero at rate y/2a 2 (C,z)log(h- d )/nh d when n — > 00 and h n < h < f) n . We make the 
following assumptions on h n , \] n and C : 

(HG 1 ) C is a VC class satisfying inf inf er 2 (C, z) =: f3 > 0. 

(HV) /i„4-0, nhil 00, log(l//t n )/log 2 n — > 00, nh d / log(l/f)„) — > 00. 

Note that (HV') is equivalent to (HV) in the specific case where p = 00. 

Theorem 3 Under assumptions (Hf), (HK1) — (HK3), (HG 1 ) and (HV 1 ), as well as 
h n = o(f)n), we have almost surely: 



log TZ n (m(C, h, z), C, h, z^j 

l l 1 I L >■> I L j / " = 1 . 

z&h, cec, log(ft-° ' 



lim sup \zZTi -d\ = (16) 

::• //. ( •• I 

The proof of Theorem [3] is provided in Section [JJ 

Remark 3 Theorem implies that, for an arbitrary e > 0, taking c = h d+e when con- 
structing confidence regions as in \12^ ensures that each m(C, h, z) belongs to its asso- 
ciated confidence interval I n (C,h,z,c). Moreover, this claim turns out to be false when 
taking c = h d ~ e with e > 0. This shows that one cannot go below the theoretical limit 
c = h d without loosing uniformity in C, h and z. 
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Remark 4 In order to obtain a confidence band for m(C, h, z) uniformly in C, h and z, 
we need the limiting distribution of 

/log(h- d ), (17) 

so Theorem 3 is not sufficient for this. Obtaining such a limit law is a real challenge in itself, 
and is beyond the scope of this paper. We leave that problem as an open problem. In the case 
of univariate kernel density estimation, Bickel and Rosenblatt (1973) showed that the supre- 
mum over the transformed kernel density estimator, obtained after a proper rescaling and a 
proper translation, converges to an extreme value distribution. The simulations in Section 
2 suggest that a proper linear transformation of [—logTZ n (m(C,h, z),C, h, z)]/ log(h~ d ) 
(depending on z,C and h) might also lead to a nondegenerate limiting distribution. 

2 Simulation results 

A simulation study is carried out to illustrate the convergence stated in ([16|) . We estimate 
the density of (|17p for four different sample sizes: n = 50, 100, 500, 1000. We specified the 
following parameters: 

1. Z is uniformly distributed on [0, 1]. Given Z = z, Y has an exponential distribution 
with expectation 1/z. 

2. C is the class of intervals [0,t], t £ [1,2]. 

3. H = [0.25,0.75]. 

4. h n = n- 1 ^- 5 and fj n = n" 1 / 54 " 5 , with 5 = 1/20. 
For each sample size, the density is estimated as follows : 

• 100 independent samples are simulated (which is enough since the density is univari- 
ate). 

• For each sample, the supremum in (|17|) is approximated by a maximum over a finite 
grid of size 50. 

• Finally, the density of (|17p is estimated by using a Parzen-Rosenblatt density esti- 
mator, applied to the 100 obtained values. We used an Epanechnikov kernel and the 
bandwidth was obtained from cross validation. 

Figure [T] shows the density estimates for n = 50, 100, 500, 1000. Figure[2]has been obtained 
from a second simulation study, where the interval [h n , f) n ] has been widened (5 = 1/10). 
As already mentioned in Remark 1.4, Figures Q] and [2] suggest that after a proper lin- 
ear transformation, the distribution of (j!7]l might converge to a non-degenerate limiting 
distribution. 



sup 



log lZ n (m(C, h, z), C, h, z 
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Density estimates for delta =1 /20 




x 



Figure 1: Estimated densities of the supremum in (|17p for S = 1/20. 

The black curve corresponds to n = 50, the light gray curve to 
n = 100, the white curve to n = 500 and the dark gray curve 
to n = 1000. 
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Density estimates for delta =1 /1 




Figure 2: Estimated densities of the supremum in (|17p for 5 = 1/10. 

The light gray curve corresponds to n = 50, the black curve to 
n = 100, the white curve to n = 500 and the dark gray curve 
to n = 1000. 
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3 Proof of Theorem [2] 



For ease of notations, we just prove Theorem [2] when k = 1. A close look at the proof 
shows that there is no loss of generality assuming k = 1. 



3.1 Truncation 

We start our proof of Theorem [2] as Einmahl and Mason did in their proof of Theorem [TJ 
As the support of K is bounded and as f) n — > we have almost surely, for all large n and 
for all z G H, g G Q, h n < h < f} n , 



Zj - z 



i=l 

E| Ic.^j + ^wjiff^) 



(18) 



where := YilQi(Zi). Hence, we can suppose that Y{ = Yilo'(Zi) without changing 
the limiting behaviour of the processes we are studying here. Now consider a sequence of 
constants (7 n )n>i fulfilling 



In 



lim inf - — , 

rc->oo (n/\og(l/h n )) l /P 



(19) 



and consider the truncated expressions, with G denoting a measurable envelope function 
of Q fulfilling (HG), 



W?(g,h,z) ^Mzy^Yl 



i=l 



c g (z)g(Y i )l {G{Yi) < ln} + dg{z))K 



Zi- z 



E 



[cg{z)g{Y i )l {G[Yi) < ln} +d g (z)^JK 



Zi - z 



(20) 



The following lemma allows us to study these truncated versions of the W n (g, h, z). 
Lemma 1 Under the assumptions of Theorem® and under h!9\) we have almost surely: 

W r J n (g,h,z)-W n (g,h,z) 



lim sup 

™^°° z&H, h n <h<t)„ 



yj2nh d \og{h- d ) 



0. 



(21) 



Proof: A careful reading of the proof of Lemma 1 in Einmahl and Mason (2000) shows 
that their assertions (2.8) and (2.9) remain true after adding a uniformity in g G Q and 
h n < h < f) n , which readily implies Lemma [1] Note also that Lemma Q] is obvious when 
(HG) is fulfilled with p = oo. □ 

The two next subsections are devoted to proving respectively the outer and inner bounds 
of Theorem [2j 
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3.2 Outer bounds 



Fix e > 0. Our goal in this subsection is to show that, almost surely 

limsup sup 7=fe= = <A(g)||^|U,2 (1 + 46). (22) 

n^oo 9 g5, z£H, h n <h<\) n \/ Znh a 10g(/i a J 

To this aim, we shall first discretise each of the sets H, [h n , f) n ] and Q into properly 
chosen finite grids, then we shall control the oscillations between elements of the grids by 
a combination of a concentration inequality which is due to Talagrand (see also Massart 
(1989), Bousquet (2002) and Klein (2002) for sharpened versions) and of an upper bound 
for the first moment of these oscillations which is due to Einmahl and Mason (2000) . 

3.2.1 Step 1: discretisations 

Consider three parameters 5\ 6 (0,1), 62 £ (0,1) and p £ (1,2) that will be chosen small 
enough in the sequel, and define the following subsequence 

n k := exp (fc/log k) ,k>5, N k := {n k -i, n k -i + 1, . . . , n k - 1}. (23) 

Note that n k /n k _i — > 1 and 

loglogn fc = log k(l + o(l)), k — > 00. (24) 

We then construct the following finite grid for each k > 1 

h nk ,R k ■= (W.! , K kt i :=p l K k , I = 0, . . . , R k - 1, (25) 

where R k := [log(fj nfc 1 //i nfe )/log(p)] + 1, and [it] denotes the only integer q fulfilling 
q < u < q + 1. Denote by | z \d'-= maxj=i ... d \ Zi \ the usual maximum norm on R . Now, 
for fixed k and < I < R k , we construct a finite grid M k ,l C H such that, given z £ H, 
there exists z G Aljt 1 fulfilling | z — z 5\h nk i. Note that one can construct this grid 
so as jj-Mfc^ < C(5ih nk: i)~ d , where C is a constant that depends only on the volume of H. 
Now set 7„ := S2[n k /log(l/h^ )) , for each > 5, n G iVjfc. By LemmaHl showing (|22l) 
is equivalent to showing that 

limsup sup / "If ' J ' <A(g)||^|| A , 2 (l + 46) (26) 

zeH, 9 eg, h n <h<t) n \J2nh a log(n a ) 

almost surely, for a proper choice of (52 > 0. 

3.2.2 Step 2: a discrete version of rt22|) 

Given a real function ^ defined on a set S 1 , we shall write: 

II i> \\s-= sup I Tp(s) I . (27) 

s£S 

Recall that, since fz is bounded away from on H, we can define 

7:= inf f z (z) >0. (28) 
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Also write, for convenience of notations 



c\\gxH-= sup \c g (z)\, \\d\\g X H-= sup \ d g (z) \ . (29) 

g&g, zGH g&g, z&H 



Our first lemma is a version of (J26J) which is discretised along the finite grids defined in 
Step 1. 

Lemma 2 For any choice of 

< d 2 < e 7 1 / 2 A(g) || K || A(2 /(6 || c || Hx0 || || Rd ), (30) 

for any finite collection {gi, . . . ,g q } C Q and for any 5\ E (0, 1), p G (1, 2), we have 

v \WZ n (gi,h nk i,z) \ s\(r\\\xr\\ /1 , ^ fon 

hmsup max — = < A{y) \ \ K \ \\2 (1 + e), (31) 

fc-^oo new fc ,i<K?, J2n fc /i d ,log(l//i d ,) 

Proof : We can assume here that q = 1 with no loss of generality. We rename in this proof 
gi to g. We define, for z £ H, h > and ~g £ G, 

^n k ,h,z,g ■ (V,X) !-»• /z(^)" 1/2 (z)j(y) l{G(j/)< 7 n fc } + d j( Z ) K ' ^ 

First note that, for each k > 5, < / < Rk and z £ Mk,l, we have 

II i>n k ,h nkil ,z.,g ll R d'xR"<( II C ||tf x g 7n fe + || d ||ff x g )7~ V2 II # ||*« 

<2 || c ||hx6 7" V2 II # Hr* ^(nfc^/logCl/O) 1 / 3 (33) 
<| || Jf ||» A(C?)K<./log(l// l l)) 1 /2 ) (34) 



where (J33J) holds for all large fc, uniformly in < I < and z € Mk,u according to 
assumption (-HV), and where (I34p holds by (!30l) . Moreover we have (recall (HK2)) 

< E 



< E (fziz)- 1 (c g (z)g(Y) + d 3 (z)) \ (|^) 

+ /z(z)- 1 || d H^ll # || 2 d p({G(y) > 7nJ n { I Z-z \ d < h na /2]) (35) 



= :j 4i iZ + ,42,z. 
The first term on the right hand side of ([35|) is equal to 



Ai+= J E^c a (z)g(Y) + d g ( Z )) 2 \z = z^^K^ Z * 



\z-z\ d <h n i/2 



l"n k ,1 



dz. 
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It follows, by making use of assumption (HC), that there exists a function r(-) fulfilling 
r(u) — >■ as u — ^ and such that 



Jz(z) 



\z-z\ d <h„ i/2 



fz{z) \h nk}l 



z — z 



dz + r(h nhj i) 



<A 2 {G)h d 



K 2 (u) — v . , ^ u(l + r(/i nfej/ )) 



[-l/2,l/2]«« 



/z(z) 



where 



e k) i := sup 

z£-ff, |«|d<l/2 



^ (l + r^,))-! 



(36) 

(37) 
(38) 

(39) 



By assumption (-ff/) and since h n , i < t)n fe _ 1 — > we readily infer that 

lim max 7 = 0. 

fc^oo 0<l<R k 

Moreover we have, uniformly in < I < Rt and z G Aifc,/ (recall (HG) and (Hf)) 

'{G(Y)> lnk }n{\Z-z\ d < h na /2}\ 



Ilk 



J ^{G 2 (Y)\Z = z)f z (z)dz 
f z [z + h nk] iu)du. 



\z-z\ d <h n i/2 



<7n>n fc ,/ « 2 



[-l/2,l/2] d 

<lnl h t k ,i a 2 || /z ||o • 



As 7„ fe — t- oo we conclude that, for all large enough k and for each < I < Rk, z £ Mk,h 
Var(Vn fel ^ fciIl z, s (^^)) < A 2 (S) || if || 2 :2 (l + e)/# fci{ . (40) 
-> R, we shall write 



Given a real function 5 : R x 



n ( 1 

^(5) := 1 9(Yi, Zi) — W.(g{Yi, Zi)j . 
i=l ^ J 



(41) 



Combining (|34"1) and (j40l) making use of the maximal version of Bernstein's inequality (see, 
e.g. Einmahl and Mason (1996), Lemma 2.2) repeatedly for each < I < Rk, z £ M-kU 
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we have, for all large k (recall that $Mk,i < G6 1 d h n d l 



| W2 n (g,h nk ,i,z) 

max — 

neN k 0^l<R k , / 2nk h d n .log(l//i* 



fc.i 



> A(0) H^lka (l + < 



Hfc> 



< 



7 jtM/c i max fi max 



r n (Vn fc A nfe , i>z>fl ) > A(g) || ^ || Ai2 (i + e ) V 2 «fc<,iiog(i/<,, 



Rk 



c 



< E ex P ( - C 1 + e2 /(l + 0) Ml A' 



1* i'- 



2C * A 



1=0 
Rk 



r >r i„de 2 /2 



(42) 



where the last inequality is a consequence of : = [log(f) rifc 1 //i nfc )/log(p)] + 1. As 
log(l/f) nfc 1 )/loglognfc_i — > oo (assumption (HV)), and by (I24p . the right hand side of 
expression (I42p is summable in k. The proof of Lemma [2] now readily follows by making 
use of the Borel-Cantelli lemma. □ 



3.2.3 Step 3: end of the proof of Theorem [2] 

Our next lemma allows us to extend the uniformity in Lemma [2] to the whole sets Q, 
\hn k i fynfc.J and H, provided that 5± > 0, 5 2 > 0, p > 1 and {g\, . . . , g q } have been properly 
chosen. Before stating our lemma, we need to recall three facts. We shall be able to properly 
discretise the class Q by making use of the following result, which is a straightforward 
adaptation of Lemma 6 of Einmahl and Mason (2000). 



Fact 1 (Einmahl, Mason, 2000) Given e > 0, there exists ho i£ > and a finite subclass 
{gi, . . . ,g q } C Q (that may depend on e) fulfilling 



sup min h~ d f z (z)E 
0<h<h , £ , ^=1, ■■■,<? 



c g (z)g(Y)+d g (z))-{c ge (z)g e (Y)+d ge (z))) K 



Z - z 



< i 



Now define the following distances on Q: 



d 2 (gug 2 ):= sup h-tfziz)- 1 ® 

0<h<ho,e, 



((c gi (z) gi (Y) + d gi (z)) - (c 92 (z)g 2 (Y) + d OT (*)))' 



x K' 



Z-z 



d(gi,g 2 ) :=m&yL{d(g 1 ,g 2 ), \\ c 9l - c g2 \\ H , \\ d gi - d 92 \\ H } ■ 



(43) 
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We write | K \ v for the total variation of K and we set, for ip : R rf i— > R, 

«vw=- -p 7rl-7r\.*>°. < 44 > 

/3i :=supE((G 2 (y) + 1) Z = z) < oo, (45) 

B :=4A || /z Holl /z 1 Ho ( II # l^d +(sup || c g \\ 2 Q Vsup || d 9 \\ 2 Q ) \ K \ 2 V ). (46) 

v ge<3 geg ' 

The following fact is a straightforward adaptation of Lemma 4 and Lemma 6 in (2000). 

Fact 2 (Einmahl, Mason, 2000) Fix e > 0. For any 5 € (0,1/2) and < h < h 0t£ 
fulfilling 

z + (2h)u £ O for each z 6 H and for each u £ M d with \ u |^< 1 (47) 

and for all large k we have, for each p £ (1, 2], z±, Z2 £ H with \ z% — z<i |^< (5h), and for 
each 51,52 G ^ fulfilling d 2 (g 1 ,g 2 ) < e, 

(Y, Z) 

<B(u 2 Cg2 (5h) V u 2 dg2 (5h) + p - 1 + 5 + e) h d . (48) 

Remarks: Assumption (I47D is just technical, in order to have the continuity arguments 
of Einmahl and Mason valid. The presence of the term p — 1 on the right hand side of 
(|48j) is due to the fact that we take care of the differences h/h nk j — 1, which are implicitly 
handled in Lemma 6 of Einmahl and Mason (2000). 

The third fact is also largely inspired by the ideas of Einmahl and Mason (2000). We remind 
that the uniform entropy number of a class of functions J- with measurable envelope F is 
defined as 

Af(e,T):= sup min {p > 1, 3(g 1 , . . . , g p ) G T 9 , sup min || g - gi ||q j2 < e || F ||q j2 }, 

Q proba geT 1=1 v,P 

where the supremum is taken over all probability measures Q. The following fact is proved 
in Varron (2008, Proposition 2.1). 

Fact 3 (Varron, 2008) Let T be a class of functions on M. d with measurable envelope 
function F satisfying, for some constants r > and h € (0, 1), 

sup Vav(g(Zi)) < r 2 h d . 

Assume that there exists 5q, C, v , (3q > and p > 2 fulfilling, for all < e < 1, 

M{e,F)<Ce~\ (49) 
^(F{Y) 2 ) </3 2 , (50) 
sup | g(z) \<5 (nh d /log{h- d ))V p . (51) 

g&T, z 
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Then there exists a universal constant A > and a parameter D{v) > depending only on 
v such that, for fixed po > 0, if h > satisfies, 

1, {4SoV^+1/t)VS=T7H , (p 5 /T 2 )^Tr P | < lQg(/t _ d) , (52) 
K 2 :=mm{l/(T 2 Po),r 2 }>h d , (53) 
i/ien we have 

F( max II T m \\ T > (t + p )D(nh d log(h~ d )) 1 / 2 ) < 4exp f- A(^) 2 log(/r rf )Y 

V l<m<n / V r / 

We can now state our second lemma, which will conclude the proof of the outer bounds 
of Theorem [2j Recall that e > was fixed at the very beginning of our proof (see Section 



Lemma 3 There exists a finite class gi,...,g q £ Q as well as two constants p e > 1 and 
Si it > small enough such that, for each 1 < p < p e and each < 5 < 5\^, we have almost 
surely : 

'W2 n (g,h nk ^z 1 )-W^(g i ,h,z 2 ) 



limsup max sup inf sup 

fc^oo ri&N k , ge gi<£<q ZuZ2eH ,\ Zl -z 2 \<6, Jln k h d ,log(l//i d ,) 

<A(G) \\ K \\ X:2 e. (54) 



Proof : 

Consider the class 



G' :={(y,s) H-Ui(c Sl (zi)5i(y)l {G(2/) < t} +d gi (z 1 ))K 



Z — Z\ 



h 

U2{c g . 2 (z 2 )g2(y)l{G(y)<t} +d g2 (z 2 ))K^ Z ^ 2 ^j, zi,z 2 € R d , 31,32 G 



*>0, (M) G (0,1) 2 , «i,«2€ [inf/ z 1/2 ,sup/ z 1/2 l) 



Recall that 7 = inf// / and note that g' admits the following function as an envelope 
function: 

G' : (y,z) ^ 2 1 ~ l ' 2 { || c \\ Hxg G(y)+ \\ d \\ Hx g ) || K \\ Rd . (55) 

Set Pi := E(G' 2 (Y,Z)) < 00 (the finiteness of /3 4 follows from (Hf) and (HG)). By an 
argument very similar to that used in Lemma 5 of Einmahl and Mason (2000) we readily 
infer that there exist C > and v > fulfilling 

M{e,g') < Ce~ v , e G (0,1]. (56) 



Recalling the notations of Fact GJ we set e = D(v)~ l (l + v / 27^Tr 1 eA(C?) || if || A)2 . By 
Fact [T] and by (HC), for any e > 0, we can choose a finite subclass {31, . . . ,g q } C G such 
that £/ is included in the finite reunion of the corresponding balls with d-radius smaller 
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than e/2. For fixed k > 5, < I < R k — 1, 1 < £ < q and 5 > 0, define the following class 
of functions: 



d(g,g e ) < e/2, h nk j <h< pK k ^. 



Obviously we always have Gk,l,£,S C Q' . By inclusion, all the classes Qk,i,i,5 inherit properties 
(j55|) and ([56p . Moreover, proving Lemma [3] is equivalent to showing that, almost surely 

,. niax ng Ar ft || T n \\g kies 

limsup max — - — < A(C/J || K ||a2 (57) 
k^oo 0<l<R k -i 2 h d \ og (i/h d ,) 

l<^<9 V nk ' 1 nk ' 1 

As h nk j < h rife _ 1 — > and by Fact [2j we can choose 5i t£ > and p e £ (1, 2) such that, for 
each 5i £ (0,Si j£ ),p £ (l,p e ), for all large A; and for all < I < Rf. — 1, 

sup h- d k J E(4, 2 {Y,Z)) <e 2 . (58) 

Recalling that h nk < h nk j < fyn k _ 1 an d assumption (HV), we can choose k large enough so 
that each class GkliSi fulfills conditions (|52p and (|53p with /3o := ,$4, h := /i nfc ,Zi n := n k> 
t := e, p := \J2/Ar and C, t> appearing in (|56p . Hence, we have, uniformly in < / < — 1 
and 1 < I < q, 



max || T n \\g KlASx> > A(G) \\ K || A , 2 e^2n k h d nkJ log(l/h d n J 



<4exp(-21og(l/^ 
Now, by Bonferroni's inequality we have, for all large k, 



V 1=0 j=i n k , 

1=0 ^ 1 ^ 1 



As log(l//i nfc )/loglog(nfc) — > oo by (HV) and (f2lj) . the proof of Lemma [3] is concluded by 
a straightforward use of the Borel-Cantelli lemma. □ 

Combining Lemmas [2] and [3] we get, for any choice of 6i, 5 2 > and p > 1 small enough, 



Wp(g,h,z) 

lim sup max sup 

k ^ n&N k , zGH> J2n k h d ,log(l/h d ,) 



<||ir|| Ai2 A(g)(l + 3e)a.s. (59) 



Now assertion (|22p is almost proved, provided that we substitute n k h d k ^ log(l//i^ fc ^) by 
nh d \og{l/h d ) in the LHS of (|59p at the minor cost of replacing l + 3e by l+4e in the RHS 
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of (|59p . This can be achieved by noticing that max{| y /n / n fc — 1 | , < n < n^} — > 

together with the following assertion 



lim sup sup 

k— >oo Kp'<P, 



I \ P hY log {{ P h)-*) 

V /i d log(/i- rf ) - £/l 



;i + 5e), 



(60) 



which, by routine computations, turns out to be true if we choose p > 1 small enough. 
This concludes the proof of the outer bounds of Theorem [2] □ 

3.3 Inner bounds 

Proving the inner bounds of Theorem [2] is a simple consequence of Theorem [TJ since, almost 
surely 



lim inf sup 



W n (g,h,z) 



> lim inf sup 



W n (g,h n ,z) 



- zeH, cec, zeH, J 2 nK\og{h 



-d\ 



A(G) \\K\\ X>2 , (61) 



where (I6ip is a consequence of Theorem [TJ 

4 Proof of Theorem [3] 

Our proof of Theorem [3] is inspired by chapter 5 in Owen (2001) and borrows some ideas 
of Chen et al. (2003). Set, for n > 1, C G C, h > and z G H, 



Zs - z 



n 

X n (C,h,z) ■=Y J K { 

n 

S n (C,h,z) ^fziz)- 1 ^ 

i=l 

w it n(C,h,z) := K ( —r 



K 



lc{Y{)-m{C,h,z)), 
Zi — z s 



h 



l c {Yi)-m(C,h,z) 



l c (Y t )-m(C,h,z) 



(62) 

(63) 
(64) 



The proof of Theorem [3] consists in showing that the quantities 

-2 log (u n (m(C, h, z),C, h,z)^j, C G C, z G H, h G [h n ,t) r 

are asymptotically equivalent to 

X n (C,h,z) 2 



U n (C,h,z) r- 



fz(z)S n (C,h,z) 



, C G C, z G H, he [h n ,\j n ], 



(65) 



and in establishing the almost sure limit behaviour of the quantities U n (C,h, z). Recall 
that a 2 (C,z) : = Var(l c (Y) | Z = z) and write 



r(C,z) :=E(l c (Y) \ Z = z). 



(66) 
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By (Hf) together with Scheffe's lemma, both cr 2 (C, •) and r(C, •) are equicontinuous uni- 
formly in C 6 C, namely 



lim sup sup | r(C, z\) — r(C, z-i) |= 0, 



lim sup sup | ct 2 (C, Z\) — <T 2 (C, 2:2) 

\zi-Z2\<S 



0. 



(67) 
(68) 



4.1 Step 1: an application of Theorem 2 

Recall that ct 2 (C, «) := Var(l c (Fi) | Z = z) and that r(C, z) := P(Y sC\Z = z). In this 
first step we prove that, given e > 0, we have (2 log(/i _d ))~ 1 L r n (C, /i, z) < (1 + e) uniformly 
in C,h,z, ultimately as n — > 00. 



Lemma 4 Under the assumptions of Theorem® we have almost surely : 

S n (C,h,z) 



lim sup 

H, ce 

h n <h<t) r < 



n ^°° z&H,c&C, nh d a 2 (C,z) \\ K \\ 2 ^ 2 



1 



0, 



lim sup 

n ^°° zeH, C&C, 

h„<h<t)„ 
As a consequence we have 



X n (C,h,z) 



2f z (z)a 2 (C,z) \\K\\ 2 X2 nh d log(h- d ) 



1. 



(69) 
(70) 



lim sup 

n ~>°° z&H.C&C, 
h n <h<i) n 



U n (C,h,z) 



2\og{h- d ) 



1 a.s. 



(71) 



Proof: 

Note that flTT]) is a consequence of flM} and ([70]). Set L(-) = K 2 (-) \\ K ||~ 2 . To apply 
Theorem [2]we write (l C (Y) - r(C,z)) 2 = 1 C (Y)(1 - 2r(C,z)) + r 2 (C,z). Notice that, 
under (HG') and (HV'), the class C and the sequence (h n ) n >i satisfy the conditions of 
Theorem [2] with p = 00. By Scheffe's lemma together with assumption (Hf) and (HG'), 
the two following collections of functions are uniformly equicontinuous on H : 



V 1 :={f z 1/2 (-)(l-2r(C,-)), CgC}, V 2 := {/, 



-1/2/ 



l (C,-),CeC}. (72) 



We can hence apply Theorem [2] to the class C, with T>i,T>2 defined as above, and with the 
kernel L to obtain, with probability one, 



lim 



sup 



W n (C,h,z) 



&h, c&c, h n <h<t) n y/2nh d log(/i~ d ) 



< 00, 



(73) 



with 



W n (C,h,z) -^fziz)- 1 ^ (lc(^)-r(C,z)) 2 L 



-E 



[lcW-riCizrfLi 
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Now write 



E{(l c (Yl)-r(C,z)) 2 L 



Zj- z 
h 



--:r(C,h,z). 



(74) 



By assumptions (HG'), (HV r ) and (Hf) together with Scheffe's lemma, we can infer that 

r(C,h,z) 



lim sup 

n ^°° zeH,ceC, 

he[h n ,t)n] 



h d f z (z)a 2 (C,z) 



1 



0, 



lim sup | m(C, h, z) — r(C, z) \= 0, 
n ^°° zeH,c&c, 

h&[h n fy n ] 

y \Qg{h- d ) 

lim sup —, — = U. 

Writing 

[s n (C,h,z)-nf z {z)- 1 \\K\\\ 2 7(C,h,z)) -W n (C,h,z) 

n 

=fz(z)- 1 [(m 2 (C, ft, z) - r 2 (C, z)) - 2(m(C, h, z) - r(C, z))] \\ K \\\ 2 J^L 

we conclude by Theorem [2] and (|76p that 

(s n (C,h,z) -nfz(z)- 1 || K \\l 2 r{C,h,z)) - \V n (C.h. :.) 
lim sup 

n ^°° zeH,c&c, 

h n <h<t] n 



(75) 
(76) 

(77) 



Zi - z 



i=i 



y / 2nh d log(h- d ) 

with probability one, from where we obtain with (|73p and (j75p that 

S n (C,h,z) -nh d || if ||| i2 <r 2 (C,z) 
lim sup 

n ^°° zeH,CeC, 
h„<h<bn 



yj2nh d \og(h~ d ) 



< CO. 



(78) 



The proof of (|69p is now concluded, by (|77p . (J7SJ) and (HG'). Assertion (|70p can be proved 
in a very similar way, taking care that the class T> := {fz(') -1 ' 2 cr(C, ^ s uniformly 

equicontinuous and bounded away from zero and infinity on H. We omit details. □ 



4.2 Step 2: convex hull condition 

The second step of our proof of Theorem [3] is usually called the "convex hull condition". 
Lemma 5 With probability one, we have, for all large n and for all C G C, z G i/, /i n < 

K(^^)(l c (^)-m(C,/ l ^)) >0} G{l,2,...,n-1}. (79) 
Proof: It is sufficient to prove that 

liminf inf pf ± (l C (Y) - m(C, h, z))k(^^) > ) > 0, (80) 
n->oo 2e //, cec, V ' \ h ) J 
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and that the following class is Glivenko-Cantelli: 

A:= G x M d , (l c (y) - m{C,h, z))K (^^) > o}, C G C, /i > 0, z G h\. 

First note that A C B, where 

B := U(y,z) G M d ' x M d , (l c (y) - a)if(^^) > o}, C G C, h > 0, z G M d , a G r|. 

By (HK1) and by Lemma 2.6.18 in Van der Vaart and Wellner (1996), the two following 
classes of sets are VC: 

B± := {{?GR rf , ±K(h- d (z- z)) > 0}, z G R d , h > o}. 

Moreover, as C is a VC class of sets, we straightforwardly deduce that the following class 
is also VC: 

Mg := {{z G X, lc(*) > a}, C G C, a G m}. 

By a combination of points (i) and (ii) of Lemma 2.6.17 in Van der Vaart and Wellner 
(1996), we conclude that B is VC, which entails that A is Glivenko-Cantelli. We now have 
to prove (j80j) . Define the following family of random variables 

H h :={(lc(Y)-m(C,h,z))K(^^j, z G H, C G C, 0< h < fj}. 

By the Cauchy-Schwarz inequality we have F(X > 0) > E(X 2 )^ 1 E(Xl x >o) 2 ■ Hence it is 
sufficient to prove that, for rj small enough we have 

inf e(x1 x> o) = - inf e( \ X \) > 0, (81) 

sup e(x 2 ) < oo. (82) 

Note that the equality appearing in (|8ip is a consequence of E(X) = for each X G Hu. 
By (HG'), (Hf) and ([67]) . routine analysis shows that, for t) small enough, both (|82"|) and 
the following assertion are true: 

inf E(X 2 ) > \ inf a 2 (C,z)f z (z) \\ K || A , 2 =: a > 0. (83) 

Now, as is uniformly bounded by some constant M > we get that ao < ME( | | ) 
for all IeW(|, and hence ([ST]) is proved. This concludes the proof of Lemma [5j □ 

4.3 Step 3: end of the proof of Theorem [3] 

Lemma [5] ensures us (see, e.g., Owen (2001), p. 219) that almost surely, for all large n and 
for each z G H, C G C, h n < h < f) n , the maximum value in 7^ n (m(C, h, z), C, h, z) is 
obtained by choosing the following weights (recall (fT3]) ): 

Pi (C,h,z) := - 1 7rir^> (84) 

n 1 + A n (C, h,z)w i>n (G, h, z) 
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where X n (C,h,z) is the unique solution of 

V Wi,n(C,M) _„ ff >- 

f^l + \ n (C,h,z) Wi , n (C,h,z) ' 1 ; 

Our next lemma gives an asymptotic control of 

sup | \ n {C,h,z) | . 

h n <h<i)n 

It is largely inspired by Lemma 1 in Chen et al. (2003). 

Lemma 6 Under the assumptions of Theorem^ we have almost surely: 



sup \ i n <l-d\ I X n(C,h,z) |= 0(1). 

h n <h<t)n 

Proof : Following the proof of Owen (2001), p. 220, Lemma |6] will be proved if we check 
the following three conditions: 



/log(/i- d ) 

max sup \ —j — \w in (C,h,z) |=o . s .(l), (87) 

l<i<« z&H, c&c, V nh d 
h n <h<t) n 

I X n (C,h,z) I 
sup = =Oo.«.(l), (88) 

zeH, CeC, ^nh d \og(h- d ) 

h„<h<t) n 

limin f in f ^n(fiM) >Qa g (89) 
/i„</i<t) n 

As each Wi n (C,h,z) is almost surely bounded by 2 || K || K d, and by (!771) . condition (I87p is 
readily satisfied. Now note that condition (|88p is a straightforward consequence of Theorem 
[2 and that (|89p is a consequence of both Lemma 3] and (HG'). The remainder of the proof 
of Lemma[6]is done by following Owen (2001), p. 220. □ 

Now set 

Vi tn (C, h, z) := X n (C, h, z)w itn (C, h, z). 
By Lemma and assertion (|87|) we have 

lim max sup | V{ n (C, h, z) |= a.s. , (90) 

rwoo l<i<n zeHt CeCj 
h n <h<t) n 
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which entails, almost surely, for all large n and for each z G H, C G C, h G [/i n , h n ]: 

n w it n(C,h,z) 
~^l + V l>n (C,h,z) 

n 

= Y, m, n ( c > h > z ){ 1 - y iAC, h, z) + V* n (C, h, z)/(l + V i>n (C, h, z)) 



" w in (C,h,z)V?(C,h,z) 
=X n (C, h, z) - f z (z)S n (C, h, z)X n (C, h,z)+Y J ' ' I (n h \ 

1 + V ijn {C, h, z) 

=X n (C, h, z) - fz(z)S n (C, h, z)X n (C, h,z)+J2 - W ^ C !^f ) ^ l(C, h, z). (91) 

J- + Vi^ n [L^, n, z) 

From (f87|) . (|88p and (|90p . we conclude that there exists a random sequence e n such that, 
almost surely, we have e n — > and 

" wf n (C,h,z) 

2^ -i r t/ . ^ W (C, /i, z) <X n (C, /i, *) max ^ (C, /i, z) 
V i,n\y, n,Z) i 

x f min | l + V i)T ,(C,/i,«) | ) \ 2 n (C,h,z) 

\ l<i<n J 

<e n ^nh d \og(h- d ), (92) 

uniformly in C € C,z £ ff, h £ [h n ,t) n ]. Hence, dividing the right hand side of (I9ip by 
S n (C, h, z), recalling (I89p and (I77p . we obtain with probability one that 

An(C, fc, z) = - f^f^ , + Pn(C, h, z), (93) 
jz(z)b n {u,h, z) 



with /3 n (C,h,z) < Me n y / log(h- d ) /nh d uniformly in C G C, z e H and /i G [h n ,t) n ], for 
some almost surely finite random variable M. We can now conclude that (recall (|65D ) 



lim sup 

n ^°° zeH,cec, 

h£[h n fy n ] 



-2 log rft„ (5, z, m(C, h, z)) ) 

U n {C,h,z) 1 



0, (94) 



by reasoning as in Owen (2001), p. 221. The proof of Theorem |3] is then concluded by 

(EU). □ 
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